Vehicle vicinity image-processing apparatus and recording medium

ABSTRACT

The present invention provides a vehicle vicinity image-processing apparatus and a recording medium capable of deducing a scene having left the field of vision of a camera mounted on the rear of a vehicle and drawing an image of the area behind the vehicle which includes this deduced part. An image-processing unit  5  first coordinate-converts a picked-up image to create a bird&#39;s-eye view image. Two chronologically consecutive bird&#39;s-eye view images are created as the bird&#39;s-eye view image, and a matching area of the two images is extracted. On the basis of this matching area, a moved area which has left the present field of vision of the camera  1  is extracted. An image of the matching area and the moved area is then drawn on a monitor  3 . By this means, it is possible to draw an image including a scene which has left the present field of vision of the camera  1.

TECHNICAL FIELD

This invention relates to a vehicle vicinity image-processing apparatusand a recording medium for picking up and displaying on a monitor animage of for example the area behind a vehicle.

BACKGROUND ART

As an apparatus for displaying the scene behind a vehicle, a displayapparatus which outputs unchanged to a monitor an image from a cameramounted on the rear of the vehicle has been known. According to thedisplay apparatus, although the situation behind the vehicle can beunderstood to some extent, there has been the problem that it isdifficult to understand the relative positions of the vehicle andobjects (for example the marking of a parking space) in the imagedisplayed on the monitor.

Separately from such a display apparatus, in Japanese Unexamined PatentPublication No. H.10-211849, technology which converts an image pickedup by a rear camera (a rear camera image) into a bird's-eye view imageand displays in this bird's-eye view image the vehicle on which the rearcamera mounted is proposed. In the case of this technology, because thevehicle is shown in the bird's-eye view image, the positionalrelationship between objects displayed in the image and the vehicle iseasier to understand than with the display apparatus which simplydisplays the picked-up image on the monitor as it is; however, there hasbeen another problem that objects outside the present field of vision ofthe camera cannot be displayed.

Consequently, when for example the vehicle is being reversed into aparking space, because the marking of the parking space which have leftthe field of vision of the camera cannot be displayed, as before, thereis the problem that maneuvering of backing the vehicle into the parkingspace is not easy.

It is therefore a first object of the present invention to provide avehicle vicinity image-processing apparatus and a recording mediumcapable of deducing and drawing a part of a scene having left the fieldof vision of for example a camera. It is a second object of theinvention to provide a vehicle vicinity image-processing apparatus and arecording medium with which it is possible to deduce and draw a part ofa scene having left the field of vision of for example a camera and alsoit is possible to lighten the computational processing load involved inso doing.

DISCLOSURE OF THE INVENTION

To achieve the above-mentioned first object, according to a first aspectof the invention, in a vehicle vicinity image-processing apparatuswhich, in a vehicle having image pick-up means (for example a camera)for picking up an image of the vehicle vicinity and display means (forexample a monitor) for displaying the image, processes the image pickedup by the image pick-up means and displays it on the display means, theimage picked up by the image pick-up means is converted into data of aground surface coordinate system projected with the image pick-up meansas a view point to create a bird's-eye view image. As a result, theimage is not a distorted image as it is picked up by for example acamera, but becomes an easily understood bird's-eye view image such asis seen from the sky above the ground surface. Next, a first bird's-eyeview image generated like this and a second bird's-eye view imagegenerated after that are compared to distinguish a matching area and anon-matching area thereof. The non-matching area obtained by thisdistinguishing is added to the second bird's-eye view image to make acomposite image, and this composite image is processed so that it can bedisplayed on the display means and is displayed on the display means.

That is, as the vehicle moves, because the field of vision of the forexample camera changes, naturally, in the first bird's-eye view imageand the second bird's-eye view image (which were picked up at differenttimes), a non-matching area, which is a difference between the images (apart which is displayed in the first bird's-eye view image but does notappear in the second bird's-eye view image), arises.

In this aspect of the invention, the difference of image between thefirst bird's-eye view image and the second bird's-eye view image isdetected, and an image of this difference part (a part which wasdisplayed before) is added to the second bird's-eye view image, wherebythe situation around the vehicle can be understood more clearly. In thisway, in this aspect of the invention, because an area having left thepresent field of vision of the for example camera can be displayed,there is the effect that for example when backing into a parking space,the maneuvering of the vehicle becomes extremely easy.

The image picked up by the image pick-up means can for example be animage of an area behind a vehicle picked up by a camera mounted on therear of the vehicle. In this case, for example when the vehicle isbacking into a parking space, because the positional relationshipbetween the marking of the parking space and the vehicle can be wellunderstood, there is the effect that parking can be carried out easily.When areas at the sides of the vehicle can also be picked up by theimage pick-up means, because when moving sideward in reverse thesituation at the sides can be well understood, sideward movement can becarried out easily. Also, besides the area behind the vehicle, anotherexample of an image that may be picked up by the image pick-up means isthe area in front of the vehicle. In this case, when images of areas atthe sides of the vehicle can also be picked up, because when movingsideward while moving forward the situation at the sides can be wellunderstood, sideward movement can be carried out easily.

Bird's-eye view image generating means may extract an image showing alane marking (of for example a parking space) in generating thebird's-eye view image. That is, when the bird's-eye view image isgenerated, although the whole of the picked-up image may be convertedinto a bird's-eye view image, alternatively just a lane marking (shownby for example white lines) maybe extracted, for example by binarizationprocessing, and displayed. In this case, because only the lane markingneeds to be processed, there is the effect that the image-processingload is lightened. When for example the vehicle is driven into a parkingspace bounded by the lane marking, only the position of the parkingspace needs to be recognized, and because when there are other things inthe image these actually make the image less clear, there is the meritthat extracting only the lane marking makes maneuvering of the vehicleeasier.

The first bird's-eye view image and the second bird's-eye view image maybe chronologically consecutive bird's-eye view images. In this case,because the difference between the first bird's-eye view image and thesecond bird's-eye view image is slight, detection of the matching areaand the non-matching area is easy. This is desirable also when thenon-matching area is being added to the second bird's-eye view image, asit ensures that there is no incongruity at the join between the images.

Preferably, display processing means performs coordinate conversion fordisplaying image data of the composite image on the display means. Thatis, to display the bird's-eye view image on for example a monitor,because adjustment of its size and so on is necessary, necessaryprocessing should be performed so that it can be displayed optimally onthe monitor.

The composite image can be compared with a third bird's-eye view imagepicked up and generated after that and a non-matching areadistinguished, and this non-matching area added to the composite imageto make a new composite image. By suitably adding a moved bird's-eyeview image in this way it is possible to synthesize images continuouslyand thereby make it possible to understand the situation around thevehicle more clearly.

To achieve the second object of the invention mentioned above, accordingto a second aspect of the invention, in a vehicle vicinityimage-processing apparatus which, in a vehicle having image pick-upmeans (for example a camera) for picking up an image of the vehiclevicinity and display means (for example a monitor) for displaying theimage, processes the image picked up by the image pick-up means anddisplays it on the display means, the image picked up by the imagepick-up means is converted into data of a ground surface coordinatesystem projected for example with the image pick-up means as a viewpoint, to create image data of bird's-eye view images successively. Thatis, image data of a first bird's-eye view image and a second bird's-eyeview image, which is a later image, are generated. As a result, theimage is not a distorted image as it is picked up for example by acamera, but becomes an easily understood bird's-eye view image such asis seen from the sky above the ground surface.

Also, in this aspect of the invention, an amount of movement of thevehicle (movement amount) is detected on the basis of vehicle signals(for example a vehicle speed signal and a yaw rate signal) obtained fromthe vehicle. Because this detected amount of vehicle movementcorresponds to an amount by which the first bird's-eye view image hasmoved, the first bird's-eye view image is moved in correspondence withthe amount of vehicle movement to create image data of a movedbird's-eye view image.

Then, image data of a new second bird's-eye view image (newer than thefirst bird's-eye view image) is combined with image data of the movedbird's-eye view image to create image data of a composite bird's-eyeview image to be displayed on the display means. That is, as the vehiclemoves, because the field of vision of (for example) the camera changes,naturally, between the first bird's-eye view image and the secondbird's-eye view image, a difference in the images (a difference in thepart picked up) arises. Accordingly, in this aspect of the invention, bysuitably adding to the second bird's-eye view image the moved bird's-eyeview image corresponding to this difference part (that is, a part whichwas in the field of vision before but is not in the present field ofvision), a part which has left the field of vision of the camera can bedisplayed on the monitor, whereby the situation around the vehicle canbe understood more clearly. In this way, in this aspect of theinvention, because an area having left the present field of vision of(for example) the camera can be displayed, there is the effect that forexample when the vehicle is being parked in reverse, this maneuveringbecomes extremely easy.

In particular, in this aspect of the invention, because the movedbird's-eye view image is not created by calculating the movement of thevehicle from changes of image with time, but rather the moved bird's-eyeview image is created using the vehicle signals, there is the merit thatthe computation for the image-processing can be simplified.Consequently, because the load on the microcomputer or the like islightened, the processing speed of the apparatus as a whole can beimproved. And because the processing can be carried out by amicrocomputer having a lower processing capacity, a contribution to costreduction is also made.

The composite bird's-eye view image can be an image in which a movedbird's-eye view image is added to the display area displayed by thesecond bird's-eye view image. This is one example of a method forforming the composite bird's-eye view image, and by this method it ispossible to display an image of the surroundings of the vehiclecontinuously. That is, because as mentioned above the image presentlybeing picked up by for example the camera can be displayed on themonitor as the second bird's-eye view image, by adding an earlier imagewhich is going to disappear from the monitor as a moved bird's-eye viewimage to the area where the second bird's-eye view image is displayed,it is possible to display on the monitor also the situation of a pastarea of the surroundings which has left the field of vision of thecamera.

As the second bird's-eye view image used in the composite bird's-eyeview image, the latest bird's-eye view image can be used. When thelatest bird's-eye view image newly picked up by the camera or the likeis used as the second bird's-eye view image, the situation of thesurroundings of the vehicle can be understood more certainly.

A bird's-eye view image made by bird's-eye view image generating meansmaybe stored in an image memory A and a composite bird's-eye view imagecomposed by composite bird's-eye view image composing means stored in animage memory B. This is an example of memories for storing the differentbird's-eye view images. And preferably, the composite bird's-eye viewimage stored in the image memory B is displayed on the display means. Inthis way, it is possible to display the composite bird's-eye view imageon a monitor or the like.

In the first and second aspects of the invention described above, whenthe composite image is displayed on the display means, an image of thevehicle maybe added to it. When the composite image showing thesituation of the surroundings of the vehicle is displayed with an imageshowing the vehicle added to it like this, because the positionalrelationship between the vehicle and for example a parking space markingbecomes clear, maneuvering of the vehicle becomes easy. Because forexample the camera is fixed in a predetermined position on the vehicle,the position of the vehicle on the screen of for example the monitor isalways the same. Therefore, the vehicle can be shown in a fixed positionon the screen of the monitor, and the vehicle may even be represented onthe screen of the monitor by printing or the like in advance. And,because the field of vision angle of the camera is also normallyconstant, not only the vehicle but also the field of vision angle of thecamera (its left-right range) may be displayed on the screen of themonitor.

Means for executing the processing of a vehicle vicinityimage-processing apparatus described above can be stored as a program ona recording medium. This recording medium may be any of variousrecording media, and may for example be an electronic control unitconstructed as a microcomputer; a microchip; a floppy disk; a hard disk;or an optical disk.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view showing the main parts of a vehicle vicinityimage-processing apparatus according to a first embodiment

FIG. 2 is a block diagram showing the electronic constitution of thevehicle vicinity image-processing apparatus of the first embodiment.

FIGS. 3A through 3C are views illustrating a processing procedurecarried out by the vehicle vicinity image-processing apparatus of thefirst embodiment;

FIG. 4 is a view illustrating positional relationship in a coordinateconversion carried out by the vehicle vicinity image-processingapparatus of the first embodiment.

FIGS. 5A through 5C are views illustrating a processing procedurecarried out by the vehicle vicinity image-processing apparatus of thefirst embodiment.

FIGS. 6A and 6B are views illustrating a procedure used for obtaining anamount by which an image has moved in the vehicle vicinityimage-processing apparatus of the first embodiment.

FIG. 7 is a flow chart of processing carried out by the vehicle vicinityimage-processing apparatus of the first embodiment.

FIG. 8 is a flow chart of processing carried out by a vehicle vicinityimage-processing apparatus of a second embodiment.

FIG. 9 is a schematic view showing the main parts of a vehicle vicinityimage-processing apparatus according to a third embodiment

FIG. 10 is a view showing the electronic constitution of the vehiclevicinity image-processing apparatus of the third embodiment.

FIGS. 11A and 11B are views showing images used by the vehicle vicinityimage-processing apparatus of the third embodiment, FIG. 11A showing animage picked up by a camera and FIG. 11B showing a bird's-eye viewimage.

FIG. 12 is a view illustrating positional relationship in a coordinateconversion carried out by the vehicle vicinity image-processingapparatus of the third embodiment.

FIG. 13 is a flow chart of processing carried out by the vehiclevicinity image-processing apparatus of the third embodiment.

FIG. 14 is a view illustrating an image composition procedure carriedout by the vehicle vicinity image-processing apparatus of the thirdembodiment.

FIGS. 15A through 15C are views showing images displayed on a monitor bythe vehicle vicinity image-processing apparatus of the third embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiments of a vehicle vicinity image-processing apparatus andrecording medium according to the present invention will now bedescribed.

First Embodiment

The basic system configuration of this embodiment will be explainedusing FIG. 1 and FIG. 2.

As shown in FIG. 1, the vehicle vicinity image-processing apparatus ofthis embodiment has a camera (for example a CCD camera) 1 mounted on therear of an automobile, an in-vehicle monitor (for example a liquidcrystal display) 3 mounted on a dashboard, and an image-processing unit5 for carrying out image-processing.

As shown in FIG. 2, the image-processing unit 5 is an electronic device,having a microcomputer as a main part, for carrying out processing ofimage data, and, functionally, it has a coordinate conversion part 11for performing a coordinate conversion on image data picked up by thecamera 1 to create a bird's-eye view image; a matching part 13 fortaking in two chronologically consecutive bird's-eye view images andcomparing them; an area deduction part 15 for deducing an area havingleft the field of vision of the camera 1 from a non-matching part of thetwo bird's-eye view images; and a drawing part 17 for drawing an imageto be displayed on the monitor 3. The image-processing unit 5 may beconstructed integrally with the camera 1.

An image processing procedure of this embodiment will now be described,on the basis of FIG. 3A through FIG. 5C. Here, the case of reversing(backing) a vehicle into a parking space will be used as an example.

An image outputted from the camera 1 (the output image) is shown in theleft side of FIG. 3A. This image is an image of the marking of a parkingspace drawn on the ground, and the vicinity of the parking space, pickedup by the camera 1; and because the camera 1 is mounted on the top ofthe rear of the vehicle, the actually rectangular parking space markingis displayed distorted in correspondence with the distances and soonbetween the vehicle (and hence the camera 1) and the positions of thelines of the parking space marking.

Next, as will be discussed in more detail later, the image on the leftside of FIG. 3A is coordinate-converted to create a bird's-eye viewimage, and then to this bird's-eye view image is added an image showingthe position of the vehicle and the field of vision angle of the camera1 to generate a composite image, and this composite image is displayedon the screen of the monitor 3 as shown on the right side of FIG. 3A.That is, because the height at which the camera 1 is mounted and itsmounting angle and field of vision angle are known, the output image ofthe camera 1 can be coordinate-converted by the coordinate conversionpart 11 (using known bird's-eye view conversion) to obtain a bird's-eyeview image like that shown in the right side of FIG. 3A.

For this processing for converting the image picked up by the camera 1into a bird's-eye view image and displaying it on the monitor 3, relatedart technology can be used (for example Japanese Unexamined PatentPublication No. H. 10-211849), as will be explained hereinafter. Here,by carrying out reverse processing of ordinary perspective conversion,the position of an image (for example a parking space marking) on aground surface is obtained as a bird's-eye view image. In more detail,as shown in FIG. 4, perspective conversion is executed in such a mannerthat position data of an image on the ground is projected onto a screenplane T which has a focus distance f from the camera position R.

Specifically, it is assumed that the camera 1 is positioned at a point R(0, 0, H) on a Z-axis, monitoring an image on a ground plane (x-ycoordinate plane) at a look-down angle τ. And accordingly, here, asshown by the formula below (1), two-dimensional coordinates (α, β) onthe screen plane T can be converted into coordinates on the ground plane(bird's-eye view coordinates) (reverse perspective conversion).

$\begin{matrix}{\begin{bmatrix}x \\y\end{bmatrix} = \begin{bmatrix}{H \cdot {\alpha/\left( {{{- \beta}\;\cos\;\tau} + {f\;\sin\;\tau}} \right)}} \\{H \cdot {\left( {{\beta\;\sin\;\tau} + {f\;\cos\;\tau}} \right)/\left( {{{- \beta}\;\cos\;\tau} + {f\;\sin\;\tau}} \right)}}\end{bmatrix}} & (1)\end{matrix}$

That is, by using the above formula (1) it is possible to convertprojected image data into image data for the screen of the monitor 3(which shows a bird's-eye view image) and display it on the monitor 3.

Next, when the vehicle backs into the parking space, as shown in theleft side of FIG. 3B and the left side of FIG. 3C, the part of theparking space marking near the vehicle progressively leaves the field ofvision of the camera 1; and at this time, the matching part 13 takes intwo chronologically consecutive bird's-eye view images and comparesthem.

Examples of consecutive bird's-eye view images (an image A changing toan image B) are shown in FIGS. 5A and 5B. As will be further discussedlater, an area which matches completely in the two images (the matchingarea) can be extracted by comparing the image A and the image B. In theimage B, because the vehicle is closer to the parking space marking thanin the case of the image A, a part of the parking space marking is cutoff. Consequently, the area deduction part 15 deduces that, of the areasof the images which do not match (the non-matching areas), the V-shapedarea near the camera 1 is an area where the image has moved (moved area)due to movement of the vehicle. And by combining the moved area of theimage A with the image B, as shown in FIG. 5C, it is possible to make acomposite image in which a part of the parking space marking outside thefield of vision of the camera 1 (the moved area) is also displayed.

Although a method in which the matching area of the image A and theimage B are extracted will be described here in after, various knownmethods can be employed for this image-processing. For example,predetermined areas (for comparing) of the images A and B can be dividedinto multiple small areas (pixels), as shown in FIGS. 6A and 6B; thepixels of the images A and B which match to the highest degree detected;and the positional relationship between the image A and the image B(i.e. how far the image A has moved) then obtained from the positionalrelationship of the matching pixels.

Specifically, as shown in FIGS. 6A and 6B, considering the case wherefor example the brightness level of the figure (or alternatively thecolor) in each pixel is expressed with a number, here, the brightnessesof the bottom three rows of pixels in the image A and the brightnessesof the top three rows of pixels in the image B match. Accordingly, inthis case, it can be supposed that the image A matches the image B bymoving through the width of one row of pixels in the vertical directionof the drawing. In this case, it appears that the figure in the image Bhas moved upward as a result of the vehicle moving downward in FIGS. 6Aand 6B.

That is, when the image A and the image B are compared, both the imagesare divided into small areas, and when the images match in most of thesesmall areas (when the degree of matching is high), the matching partsare regarded as the same image. Accordingly, when in the image A and theimage B the matching parts are shifted, it can be inferred that theyhave become different due to the image having moved, and the amount andthe direction of that movement can also be obtained. Similarly, amountsof movement in the left-right direction and diagonal directions can alsobe obtained.

When this composite image is displayed on the monitor 3, as shown in theright side of FIG. 3B and the right side of FIG. 3C, the drawing part 17draws the composite image and also draws an image showing the positionof the vehicle and the field of vision angle of the camera 1. In thiscase, because the camera 1 is fixed to the vehicle in an unchangingposition, the image of the vehicle and the field of vision angle of thecamera 1 can be displayed in the same position on the screen of themonitor 3 at all times (for example the upper part of the center of thedisplay screen).

Processing carried out by the image-processing unit 5 and so on will nowbe described, on the basis of the flow chart of FIG. 7.

As shown in FIG. 7, first, instep 100, it is determined whether or notthe gear shift position is Reverse. When an affirmative determination ismade here processing proceeds to step 110, and when a negativedetermination is made processing proceeds to step 170. The shiftposition can be detected on the basis of a signal from a shift positiondetection sensor (not shown) or a signal from another electronic controlunit.

In step 170, the image of an area outside the field of vision of thecamera drawn previously is erased; that is, only the present cameraimage (bird's-eye view image) is displayed, and processing returns tostep 100.

In step 110, on the other hand, coordinate conversion (theabove-mentioned bird's-eye view conversion) of the image picked up bythe camera 1 is carried out, and a bird's-eye view image is obtained. Inthe following step 120, a matching area of two consecutive bird's-eyeview images is extracted. For example from the image A and the image Bin FIGS. 5A and 5B, the matching area is extracted. And in the followingstep 130, a moved area outside the present field of vision of the camera1 is extracted.

In step 140, whereas the moved area of the image A is later to be drawnoutside the field of vision of the camera in the image B, before it isdrawn, the image B already drawn is shifted by an amount correspondingto the moved area (downward in the image B), whereby a drawing area issecured.

In the following step 150, as shown for example in FIG. 5C, the movedarea is drawn in the drawing area secured in the foregoing step 140 (acomposite image is composed). And in the following step 160, as shownfor example in the right side of FIG. 3C, the vehicle and a V-shapedline showing the field of vision angle are drawn, and the presentprocessing ends.

Preferably, when a composite image has already been composed and stored,this composite image and the latest bird's-eye view image are comparedand a non-matching area thereof is identified, and this non-matchingarea is added to the stored composite image to make a new compositeimage. In this way it is possible to synthesize images continuously bysuitably adding each post-movement bird's-eye view image to thecomposite image, and as a result it is possible to understand thesituation around the vehicle more clearly.

Thus, in this embodiment, an image picked up by a camera is converted toa bird's-eye view image; a matching area of chronologically consecutivebird's-eye view images is extracted; and a moved area, which is adiffering part of the two bird's-eye view images, is extracted. Then,the moved area is added to the bird's-eye view image showing the presentimage to make a composite image, and when the composite image isdisplayed on the screen of the monitor 3, an image of the vehicle andthe field of vision angle of the camera are added. That is, in thisembodiment, when the vehicle moves, an area having left the field ofvision of the camera is deduced and combined with the present cameraimage, whereby it is possible to display the area which is outside thefield of vision of the camera and ordinarily could not be displayed; andconsequently, it is possible to understand the positional relationshipbetween the vehicle and the parking space clearly. As a result, there isthe valuable effect that maneuvering such as reversing into a parkingspace becomes extremely easy.

Second Embodiment

A second embodiment of the invention will now be described, althoughparts the same as parts in the first embodiment will not be explainedagain here. In this embodiment, only image of a parking space markingand car stoppers and the like is extracted and drawn, and the image ofthe background is dispensed with.

As shown in the flowchart of FIG. 8, first, in step 200, it isdetermined whether or not the gear shift position is Reverse. When anaffirmative determination is made here processing proceeds to step 210,and when a negative determination is made processing proceeds to step280.

In step 280, a previously drawn image of an area outside the field ofvision of the camera is erased. In step 210, on the other hand, an imageof the parking space marking and car stoppers and the like is extractedby means of binarization processing from the image picked up by thecamera. Here, it is assumed that the parking space marking and the carstoppers are painted in white and are easily distinguishable from aground surface of black asphalt.

In the following step 220, the binarized image is subjected tobird's-eye view conversion to create a bird's-eye view image of theparking space marking and the car stoppers. Here, the background is of asingle color. Then in step 230, a matching area of two consecutivebird's-eye view images is extracted. And in step 240, a moved areahaving left the present field of vision of the camera 1 is extracted.

In the following step 250, before an image of the moved area is drawn,the image B already drawn is shifted (downward in the image B) by theamount of the moved area to secure a drawing area. Then in step 260, animage of the moved area is drawn in the drawing area secured in step 250(a composite image is composed). Instep 270, the vehicle and a V-shapedline showing the field of vision angle of the camera are drawn, and theprocessing routine ends.

In this embodiment also, the same benefits as those of the firstembodiment are obtained; moreover, in this embodiment, the entirepicked-up image is not converted to a bird's-eye view image anddisplayed, but rather just things necessary for parking, for exampleonly the parking space marking and car stoppers, besides the vehicle andthe field of vision angle, are displayed. As a result, the imageprocessing load is lightened; the image displayed on the monitor 3becomes easy to watch; and there is the merit that maneuvering of thevehicle becomes easy.

Third Embodiment

A third embodiment will now be described. First, the basic systemconfiguration of this embodiment will be explained, using FIG. 9 andFIG. 10.

As shown in FIG. 9, the vehicle vicinity image-processing apparatus ofthis embodiment has a camera (for example a CCD camera) 21 mounted onthe rear of the vehicle; an in-vehicle monitor (for example a liquidcrystal display) 23 mounted on the dashboard of the vehicle; a vehiclespeed sensor 25 for detecting a vehicle speed; a yaw rate sensor 27 fordetecting a yaw rate; and an image-processing unit 29 for carrying outimage-processing.

The image-processing unit 29, as shown in FIG. 10, is an electronicdevice, having a microcomputer as a main part, for carrying outprocessing of image data, and has a coordinate conversion part(bird's-eye view image conversion circuit) 31 for performing coordinateconversion of image data picked up by the camera 21; an image memory Afor temporarily storing image data of a created bird's-eye view image; avehicle speed signal input circuit 33 for inputting a vehicle speedsignal; a yaw rate signal input circuit 35 for inputting a yaw ratesignal; a CPU 37 for reading in the vehicle speed signal and the yawrate signal and carrying out computation such as that for moving thebird's-eye view image (for example turning it); and an image memory Bfor storing data of a bird's-eye view image to be displayed on themonitor 23.

Here, as will be further discussed later, the image data of a bird's-eyeview image stored in the image memory B is image data of a bird's-eyeview image (composite bird's-eye view image) made by combining an image(moved bird's-eye view image) obtained by moving a bird's-eye view image(first bird's-eye view image) created on the basis of image data pickedup by the camera 21 at the previous time (for example at a time T) witha bird's-eye view image (second bird's-eye view image) created on thebasis of image data picked up by the camera 21 newly (for example at atime T+1) The image-processing unit 29 may be constructed integrallywith the camera 21, and the constituent parts of the image-processingunit 29 may be partially or entirely integrated by an LSI.

A procedure of image processing in this embodiment will now bedescribed, on the basis of FIGS. 11A and 11B and FIG. 12. The case ofreversing (backing) a vehicle into a parking space will be used as anexample.

First, a procedure by which an image inputted from the camera 21 isconverted into a bird's-eye view image and stored in the image memory Awill be explained. An unprocessed image (output image) outputted fromthe camera 21 is shown in FIG. 11A. Because this unprocessed image is animage of the marking of a parking space drawn on the ground and thevicinity thereof, picked up by the camera 21 mounted on the top of therear of the vehicle, the actually rectangular parking space marking aredisplayed distorted in correspondence with the distances and so onbetween the vehicle (and hence the camera 21) and the positions of eachline of the parking space marking.

Accordingly, by the same method as that described in the firstembodiment, the image data of the unprocessed image iscoordinate-converted to generate a bird's-eye view image withoutdistortion, like that shown in FIG. 11B, and the image data of thisbird's-eye view image is stored in the image memory A.

Next, a procedure by which, using the vehicle speed signal and the yawrate signal, the bird's-eye view image stored in the image memory A ismoved (for example turned) in correspondence with the amount by whichthe vehicle has moved (movement amount) will be explained. Here, in FIG.12 the position of the vehicle at a time T is shown, and a distancethrough which the vehicle moves by a time T+1 is written L and an anglethrough which the vehicle moves by the time T+1 is written θ1 (=turnangle θC at turning center).

First, when the sampling time of the camera 21 is 100 [ms], using thevehicle speed S [km/h] obtained from the vehicle speed signal, thedistance L [m] through which the vehicle has moved can be calculatedfrom the following formula (2).L[m]=S[km/h]×1000[m/km]÷3600[S/h]×100[ms]÷1000[ms/s]  (2)

And, because there is the relationship of the figure shown in FIG. 12,using the yaw rate θ0 [°/s] obtained from the yaw rate signal, the anglethrough which the vehicle has moved can be calculated from the followingformula (3).θ1[°]=θ0[°/s]×100 [ms]÷1000[ms/s]  (3)

Also, here, because it is being assumed that the turning center of thevehicle is on the extension line of the rear axle, if the forwarddirection of the vehicle is written Y and the direction perpendicular tothis is written X, computing with the camera position as the X-directionorigin, the turning center position XC [m] in the X-direction can becalculated from the following formula (4).XC[m]=L[m]÷TAN(θ1)  (4)

On the other hand, the turning center position in the Y-direction is atthe rear axle, and the distance from the camera 21 to the rear axle isYC [cm]. And, from the relationship of the figure shown in FIG. 12, theturn angle θC at the turning center is the same as the angle θ1 throughwhich the vehicle has moved.

Next, using the turning center position (XC, YC) and turn angle θCobtained as explained above, the bird's-eye view image is moved asfollows.

The image data of the bird's-eye view image stored in the image memory Ais also stored once in the image memory B. Accordingly, here, the imagedata of the bird's-eye view image stored in the image memory B istransferred to internal memory of the CPU 37, and the bird's-eye viewimage is turned using the turning center position (XC, YC) and the turnangle θC.

A conversion formula for carrying out this turning movement is shownbelow as formula (5), and by internal processing of the CPU 37 usingthis formula, with respect to post-movement coordinates (XB, YB)pertaining to after the move through the turn angle θ, correspondingpre-movement points (XA, YA) are calculated, and (XA, YA) pixel valuesare saved in the (XB, YB) address memory.XA=XB·COS θ+YB·SIN θYA=YB·COS θ−XB·SIN θ  (5). . . where (XA, YA) are pre-movement coordinates; (XB, YB) arepost-movement coordinates; and θ is the turn angle.

Then, by performing these calculations for all the coordinates, from thebird's-eye view image stored in the image memory A (the first bird's-eyeview image, at the time T), a moved bird's-eye view image (movedbird's-eye view image at the time T) is created. And along with this,the bird's-eye view image created on the basis of the image newly pickedup by the camera 21, that is, the present bird's-eye view image (thesecond bird's-eye view image, at the time T+1) is newly transferred fromthe image memory A to the CPU 37.

And accordingly, in the CPU 37, the second bird's-eye view image iswritten to a position where the positional relationship corresponds withthe time T (a position where there is no image misalignment with themoved bird's-eye view image), and by this processing the movedbird's-eye view image and the second bird's-eye view image are combinedto create image data of a composite bird's-eye view image and this imagedata is stored in the image memory B.

In the creation of the composite bird's-eye view image, although animage of the position of the vehicle and the field of vision angle ofthe camera 21 may be added and stored in the image memory B and thiscomposite bird's-eye view image displayed on the screen of the monitor23, alternatively, instead of an image of the position of the vehicleand the field of vision angle of the camera 21 being stored in the imagememory B, an image showing the position of the vehicle and the field ofvision angle of the camera 21 may be added when the composite bird's-eyeview image stored in the image memory B is drawn to the monitor 23.

Next, processing carried out by the image-processing unit 29 will bedescribed, on the basis of the flow chart of FIG. 13.

As shown in FIG. 13, first, in step 300, it is determined whether or notthe gear shift position is Reverse. When an affirmative determination ismade here processing proceeds to step 310, and when a negativedetermination is made processing proceeds to step 390. The shiftposition can be detected on the basis of a signal from a shift positiondetection sensor (not shown) or a signal from another electronic controlunit.

In step 390, any previously drawn image of an area outside the field ofvision of the camera 21 is erased; that is, only the present image fromthe camera 21 (bird's-eye view image) is displayed; and processing thenreturns to step 300. In step 310, on the other hand, coordinateconversion (bird's-eye view conversion) of the image picked up by thecamera 21 is carried out, and a bird's-eye view image is created.

In step 320, this bird's-eye view image is stored in the image memory A.At this time, the same bird's-eye view image is also stored in the imagememory B. In step 330, on the basis of the vehicle speed signal and theyaw rate signal, an amount of movement of the vehicle (movement amount),expressed by a movement distance L and a turn angle θC, is obtained.

In the following step, step 340, on the basis of the amount of movementof the vehicle, the bird's-eye view image stored in the image memory B(first bird's-eye view image, at the time T) is moved, and apost-movement bird's-eye view image (moved bird's-eye view image fromthe time T) is obtained.

In step 350, a second bird's-eye view image (at the time T+1), newlypicked up and stored in the image memory A, is read in from the imagememory A. And in step 360, the moved bird's-eye view image and thesecond bird's-eye view image are combined to generate a compositebird's-eye view image. Specifically, as shown in FIG. 14, the compositebird's-eye view image is composed by writing to outside the new field ofvision of the camera (corresponding to the second bird's-eye view image)a part having left the field of vision of the camera due to the latestmovement of the vehicle; that is, a part of the moved bird's-eye viewimage made by moving the first bird's-eye view image in correspondencewith the amount of movement of the vehicle (specifically, the part whichis left after a part overlapping with the second bird's-eye view imageis removed). In FIG. 14, the vehicle is shown for reference; inpractice, a frame or the like showing the vehicle is drawn when thecomposite bird's-eye view image is drawn on a screen of the monitor.

In the following step 370, the composite bird's-eye view image is storedin the image memory B. In step 380, an image showing the vehicle and animage showing the field of vision angle are added; the compositebird's-eye view image stored in the image memory B is displayed on themonitor 23; and the present processing routine ends.

In this way, in this embodiment, the image picked up by the camera 21 isconverted into a bird's-eye view image; an amount by which the vehiclehas moved is detected on the basis of vehicle signals; and incorrespondence with this vehicle movement amount the bird's-eye viewimage is moved to create a moved bird's-eye view image. Then, the movedbird's-eye view image and a newly read-in bird's-eye view image arecombined to compose a composite bird's-eye view image; images of thevehicle and the field of vision angle are added; and this compositebird's-eye view image is displayed on the screen of the monitor 23.

That is, in this embodiment, when the vehicle moves, by deducing an areahaving left the camera view and combining it with the present cameraview, it is possible to display parts which are outside the field ofvision of the camera and ordinarily could not be displayed; andconsequently, as shown in FIGS. 15A, 15B and 15C, it is possible tounderstand the positional relationship between the vehicle and theparking space clearly. As a result, there is the marked benefit thatmaneuvering such as reversing into a parking space becomes extremelyeasy.

In FIGS. 15A, 15B and 15C, display pictures of the monitor 23 producedby the method of this embodiment are shown in order, when a vehiclefirst turns and then moves straight while backing into a parking spaceare shown in order.

In particular, in this embodiment, because the moved bird's-eye viewimage is created using vehicle signals obtained from a vehicle speedsensor 25 and a yaw rate sensor 27, rather than being created forexample by calculating a movement of the vehicle from change of imagewith time between first and second bird's-eye view images, there is themerit that the computational processing for this is easy. That is,because image-processing such as matching of images, which necessitatesa large amount of computation, can be dispensed with, the processingload is lightened and the necessary image can be displayed on themonitor 23 swiftly.

Consequently, the load on the microcomputer and so on is lightened, andthe processing speed of the apparatus as a whole can be increased.Because the processing can be carried out by a microcomputer with alower processing capacity, a contribution to cost reduction is alsomade.

The present invention is not limited to the embodiments described above,and various changes can of course be made without deviating from thescope of the invention.

(1) For example, in the first embodiment, preferably, the picked-upimage of the parking space marking and the car stoppers and so on is notjust simply coordinate-converted and displayed, but rather an image ofthe parking space marking and the car stoppers, which serve as guidesfor maneuvering the vehicle, is extracted from the image picked up bythe camera, and the parking space marking and the car stoppers in theimage are strengthened in color and/or tone.

(2) And, in the third embodiment, although in the above-mentionedprocessing was started when it was detected that the shift position wasReverse, alternatively, so that processing is carried out also when thevehicle has switched to moving forwards after a reversing movement,starting and continuing of processing may alternatively be decided onthe basis of a vehicle signal. For example, processing may be startedwhen the vehicle speed is 10 km/h, and processing executed to display abird's-eye view image for as long as the vehicle speed remains belowthis speed. By this means it is possible to display a bird's-eye viewimage continuously, without being reset, even if the shift positionchanges in the course of a parking operation.

(3) Also, as another application of the invention, by means of a cameraviewing an area in front of a vehicle, it is also possible to display acomposite bird's-eye view image of the proximity of the vehicle bodyincluding the area having left the field of vision of the camera from infront of the vehicle, and thereby provide information by which it can bejudged whether or not it is possible to squeeze through a narrow streetor the like.

(4) And, whereas in the first, second and third embodiments a vehiclevicinity image-processing apparatus was discussed, the invention alsoprovides a recording medium on which is recorded means for executing theprocessing of such an apparatus. This recording medium may be any ofvarious recording media, and may for example be an electronic controlunit constructed as a microcomputer; a microchip; a floppy disk; a harddisk; and an optical disk. That is, it maybe any recording medium onwhich is recorded means, such as for example a program, capable ofexecuting the processing of a vehicle vicinity image-processingapparatus described above.

INDUSTRIAL APPLICABILITY

As described above, a vehicle vicinity image-processing apparatus and arecording medium according to the present invention can draw on amonitor an image of the vicinity of a vehicle including a deduced parthaving left the field of vision of a camera, and accordingly they arehighly suitable for use for example in equipment for assisting drivingoperation such as parking.

1. A vehicle vicinity image processing apparatus which is applied to avehicle having image pick-up means for picking up an image of a vehiclevicinity and display means for displaying an image, and which processesthe image picked up by said image pick-up means and displays theprocessed image on said display means the vehicle vicinityimage-processing apparatus comprising: bird's eye view image creationmeans for creating a bird's eye view image on the basis of the imagepicked up by said image pick-up means; vehicle movement amount detectingmeans for detecting an amount of vehicle movement on the basis ofvehicle signals obtained from the vehicle; bird's-eye view image movingmeans for moving a first bird's-eye view image created by saidbird's-eye image creation means in correspondence with the amount ofvehicle movement and thereby creating a moved bird's-eye view image;composite bird's-eye view image creation means for creating a compositebird's-eye view image to be displayed on said display means by combininga second bird's-eye view image which is picked up and created later thanthe first bird's-eye view image with the moved bird's-eye view image,wherein said composite bird's-eye view image is an image in which a partof the moved bird's-eye view image that does not overlap with the secondbird's-eye view image is added to a display area of the secondbird's-eye view image.
 2. A vehicle vicinity image-processing apparatusaccording to claim 1, wherein said vehicle signals include a vehiclespeed signal and a yaw rate signal.
 3. A vehicle vicinityimage-processing apparatus according to claim 1, wherein the secondbird's-eye view image used in creating the composite bird's eye viewimage includes a latest bird's-eye image.
 4. A vehicle vicinityimage-processing apparatus according to claim 1, further comprising: afirst memory for storing the bird's-eye view image created by saidbird's-eye view image creation means; and a second memory for storingthe composite bird's-eye view image created by said composite bird's-eyeview image creation means.
 5. A vehicle vicinity image-processingapparatus according to claim 4, wherein the composite bird's-eye viewimage stored in said second memory is displayed on said display means.6. A vehicle vicinity image-processing apparatus according to claim 1,wherein an image showing the vehicle is added to the compositebird's-eye view image when the composite bird's-eye view image isdisplayed on said display means.
 7. A vehicle vicinity image-processingapparatus according to claim 1, wherein, after a gear shift position isdetermined to be Reverse, processing carded out by each of saidbird's-eye view image creation means, vehicle movement amount detectingmeans, bird's-eye view image moving means, and composite bird's-eye viewcreation means is started.
 8. A vehicle vicinity image-processingapparatus according to claim 1, wherein said image pick-up meansincludes a camera for viewing an area in front of the vehicle.
 9. Acomputer-readable recording medium storing a program for executing aprocessing of the vehicle vicinity image-processing apparatus of claim1.